Minimax and the Value of Information Evan Sadlery February 13, 2013 Abstract In his discussion of minimax decision rules, Savage (1954, p. 170) presents an example purporting to show that minimax applied to negative expected utility (referred to by Savage as \negative income") is an inadequate decision criterion for statistics; he suggests the application of a minimax regret rule instead. The crux of Savage?s objection is the possibility that a decision maker would choose to ignore even \extensive" information. More recently, Parmigiani (1992) has suggested that minimax regret su ers from the same aw. He demonstrates the existence of \relevant" experiments that a minimax regret agent would never pay a positive cost to observe. On closer inspection, I nd that minimax regret is more resilient to this critique than would rst appear. In particular, there are cases where no experiment has any value to an agent employing the minimax negative income rule, while we may always devise a hypothetical experiment that a minimax regret agent would pay for. The force of Parmigiani?s critique is further blunted by the observation that \relevant" experiments exist for which a Bayesian agent would never pay. I conclude by discussing the notion of pessimism in the context of minimax decision rules. 1 Introduction In recent years there has been a revival of interest in the application of minimax procedures to statistical and econometric problems. Manski (2004) for instance, has spawned a literature on the application of minimax regret to treatment choice. One reason for this newfound popularity is surely the increase in computing power now available to researchers, rendering such methods feasible in more domains. With renewed interest has come increased scrutiny. In particular, concerns over the use of information by minimax procedures have resurfaced. It has long been known that minimax applied to negative expected utility (hereafter, \negative income") can entirely ignore available information in some decision problems. Savage (1954, p. 170) presented a simple example demonstrating this phenomenon, asserting that minimax applied to negative income was entirely inadequate as a criterion for statistics. His alternative was applying minimax to regret. While many credit Savage with the invention of the minimax regret criterion, Savage himself gave priority to Wald (1950), believing that Wald could not possibly have intended minimax negative income in his work. Savage clearly believed this critique inapplicable to regret, but more recent work has disputed this. Parmigiani (1992) analyzes an example in which minimax applied to negative income assigns positive value to an experiment, while minimax regret does not. He goes on to characterize the circumstances under which minimax rules refrain from \relevant" experimentation, even if the experiment is nearly costless. Parmigiani describes this phenomenon as ultrapessimism. The essence of his result is best understood by interpreting the statistician?s problem as a zero sum game played against nature. If nature has an equilirium strategy that is supported on relatively few states, then experiments that fail to distinguish these states will be worthless. If such cases can be expected to occur regularly outside of contrived examples, then this raises serious concerns about the use of minimax regret in applied work. I am grateful to Roy Radner for calling my attention to this problem as well as for numerous fruitful discussions. I also thank J org Stoye for pointing me to useful references. Any errors are mine alone. yStern School of Business, New York University { esadler@stern.nyu.edu 1 One goal of the present paper is to controvert this criticism of minimax applied to regret. Savage?s example uncovered a more fundamental aw in minimax negative income than that discussed by Parmigiani. In some decision problems, minimax negative income not only fails to utilize an observation that another procedure nds valuable, but in fact, no conceivable experiment provides any value to an agent following a minimax negative income rule. This is in stark contrast to the situation we nd with minimax regret, where a valuable experiment can always be constructed. Moreover, even when valuable experiments do exist for the minimax negative income rule, there may still be some states such that no amount of evidence in their favor will cause the decision maker to adopt an action better suited to those states. Again, minimax applied to regret avoids such issues. On further examination, the characterization of ultrapessimism given by Parmigiani can apply to Bayesian procedures just as easily|even when a minimax regret agent would pay for the information. Ultrapessimism, so de ned, fails to distinguish meaningfully between decision procedures, but Savage?s observation remains a substantive distinction between minimax regret and minimax negative income. A second goal is to critically examine other objections to minimax regret. There are other senses in which the procedure can be described as \pessimistic," and these may present challenges to researchers in applications. Many of the issues raised stem from prescriptions that go against what we intuitively expect from a \good" estimator. In a sense, regret is criticized for failing to conform to our prior information, but a major motivation for minimax procedures is to avoid subjective priors. An important direction for future work lies in devising ways to utilize such information alongside minimax procedures. In the next section I provide a detailed analysis of the examples given by Savage (1954) and Parmigiani (1992). The following section generalizes previous results that characterize when minimax values information. This section also establishes the key distinction between minimax applied to negative income and minimax applied to regret. Finally, I o er some thoughts on the problem of \pessimism" in minimax estimation and the e ective application of minimax regret. 2 Examples In this section I present two examples; rst, Savage?s (1954) example, followed by Parmigiani?s (1992). Consider a state space partitioned into two events, B1 and B2, with two primary acts, f1 and f2. Suppose f1 yields a payo of 1 utiles in both events, and f2 yields 10 if B1 obtains and 1 if B2 obtains. B1 B2 f1 1 1 f2 10 1 Assuming the decision maker is capable of making randomized choices, there are derived acts of the form f = f1 + (1 )f2 for 2 [0; 1]. The negative income from a particular act in each event is then, I(f ;B1) = 10 9 ; I(f ;B2) = 2 1: Note that for all values of in the given range, I(f ;B1) is larger than I(f ;B2), so if our decision maker applies a minimax negative income decision rule, the chosen act will be the one that minimizes I(f ;B1), which is to simply choose f1 with probability 1. Now suppose the decision maker has the option to observe at no cost the random variable X taking values in f1; 2g with P(X = 1 jB1) = P(X = 2 jB2) = 1 g; P(X = 2 jB1) = P(X = 1 jB2) = g; for some g < 12 . There are now four primary acts as functions of X: g1(X) = f1 for each value of X, g2(X) = f2 for each value of X, g3 de ned by g3(1) = f1 and g3(2) = f2, and g4 de ned by g4(1) = f2 and g4(2) = f1. The set of derived acts available to the decision maker are the convex combinations of these four: f = 1g1 + 2g2 + 3g3 + 4g4, where 1 + 2 + 3 + 4 = 1. For such an act, the negative income in each event can be calculated as, 2 I(f ;B1) = 1 + 10 2 + (1 + 9g) 3 + (10 9g) 4; I(f ;B2) = 1 2 + (2g 1) 3 + (1 2g) 4: Note that just as before, I(f ;B1) I(f ;B2) given the constraints on the parameters. In the expression for I(f ;B1), the coe cients of the 2, 3, and 4 terms are all strictly larger than 1, so this is minimized by allocating all weight to 1. That is, the decision maker chooses f1 regardless of the observed X and regardless of the value g, earning utility of 1 in all cases. If observing X were costly, the agent would evidently decline the observation, no matter how small the cost. This is the basis of Savage?s criticism of the minimax negative income rule. Compare this result with that obtained by a decision maker using a minimax regret rule when faced with the same situation. Let L(f ;Bi) denote the regret of action f if Bi obtains. For the original problem in the absence of the observation, L(f ;B1) = 9(1 ); L(f ;B2) = 2 : The decision maker chooses to minimize the maximum of L(f ;B1) and L(f ;B2). The two are equal when = 911 , and since L(f ;B1) is a strictly decreasing function of and L(f ;B2) is a strictly increasing function of , this gives the minimum. The regret at the minimum is 1811 . Given the same random variable X as before, and the same four primary acts, the situation changes. The regret in each event is, L(f ;B1) = 9 2 + 9g 3 + 9(1 g) 4; L(f ;B2) = 2 1 + 2g 3 + 2(1 g) 4: Note that since g < 12 , both of these are reduced by reallocating weight from 4 to 3, so we can take 4 = 0. Now consider the following observations. In the solution we must have 1 (9 2 + 7g 3)=2, otherwise the maximum can be reduced by reallocating weight from either 2 or 3 to 1. If 1 (9 2 + 7g 3)=2, then L(f ;B1) L(f ;B2). Further, the solution must have L(f ;B1) = L(f ;B2) since if L(f ;B1) < L(f ;B2), the maximum can be reduced by reallocating weight from 1 to 2. Write 3 = 1 1 2, and this condition becomes (9 9g) 2 9g 1 + 9g = (2 2g) 1 2g 2 + 2g =) 1 = (9 7g) 2 + 7g 2 + 7g : The regret is then (9 9g) 2 9g (9 7g) 2 + 7g 2 + 7g + 9g = 18 36g 2 + 7g 2 + 9g + 63g2 2 + 7g : Since g < 12 , the coe cient of 2 is positive, and the regret is minimized by choosing 2 = 0. Thus, the minimax regret act sets 1 = 7g=(7g + 2), 2 = 0, 3 = 2=(7g + 2). The regret is L(f ;B1) = L(f ;B2) = 18g=(7g + 2), which goes to zero as g goes to zero (i.e. as X becomes more informative). The agent should be willing to pay as much as 1811 18g=(7g + 2) > 0 to observe X. For the second example, consider a box containing two ordered marbles in one of three arrangements. Either both are red, the rst is blue with the second red, or both are blue. Two primary acts are available. The act f1 pays 2 utiles if both are red, 0 if the rst is blue and the second red, and 4 if both are blue. The act f2 pays 4 if both are red or the rst blue and the second red, and it pays 0 if both are blue. The payo table is depicted below. RR BR BB f1 2 0 4 f2 4 4 0 3 Considering mixed actions as before, the negative income from choosing f = f1 + (1 )f2 in each state is I(f ;RR) = 4 2 ; I(f ;BR) = 4 4 ; I(f ;BB) = 4 : The minimax solution sets = 23 , giving expected negative income of 8 3 in all states. If the decision maker can observe the rst marble at no cost, there are four primary acts as a function of the observation. In fact, we need only consider two of these acts since f1 dominates f2 whenever the observed marble is red. If the rst marble is blue, then clearly the minimax solution is to choose f1 and f2 with equal probability, leading to expected negative income of 2 in each state. The decision maker should be willing to pay as much as 23 to observe the rst marble. Now consider a decision maker using minimax regret in the same situation. The regret table is RR BR BB f1 0 0 4 f2 2 4 0 The regret from the act f = f1 + (1 )f2 in each state is L(f ;RR) = 1 ; L(f ;BR) = 2 2 ; L(f ;BB) = 2 : The minimax regret strategy is to randomize equally between the two acts, yielding a maximum expected regret of 2. Given the opportunity to observe the rst marble, a cursory examination of the payo table shows that zero regret can be achieved in state RR, but will be unchanged in the other two. The optimal strategy will be to randomize equally between the act that chooses f1 always and the act that chooses f1 only when the observed marble is red. The maximum regret is still 2, so the decision maker should be unwilling to pay any positive amount for the observation based on the minimax regret criterion. Parmigiani (1992) claims that this example shows minimax regret is vulnerable to the same criticism as minimax negative income. In the next section, I shall rebut this claim while providing some insight into when these two decision rules will ignore available information. 3 The Value of an Experiment The phenomenon demonstrated by these examples can be well-understood through the correspondence between the decision maker?s problem and a two player zero sum game played between nature and the decision maker (Wald, 1950). Consider a decision problem with a nite collection of states and a nite number of primary acts. Imagine the corresponding payo table, either a negative income or a regret table. If this is considered as a zero sum game where nature earns the negative of the decision maker?s payo , then it is well-known that the Nash equilibria of the game share a common value, and this is the value the decision maker will achieve through application of a minimax decision rule. Let = ( d; n) denote an equilibrium strategy pro le in the game described above, and let denote the collection of states that nature plays with positive probability under . Consider a random variable X taking values in some arbitrary measure space (S;S); we shall call such a random variable an experiment when the decision maker can observe its value prior to acting. We obtain the following result. Proposition 1. In the decision problem described above, the experiment X has no value to a minimax decision maker if there exists an equilibrium strategy pro le such that P(X 2 S j!1) = P(X 2 S j!2) for all !1; !2 2 and for all S 2 S. If there are only two primary acts, and the experiment X has no value, then such an equilibrium strategy pro le necessarily exists. 4 Proof. Observe rst that the value of the game to the decision maker can be no lower with the experiment since the decision maker may always choose to ignore it. If we have P(X 2 S j!1) = P(X 2 S j!2) for all !1; !2 2 and all S 2 S, then nature may play the strategy speci ed by and be sure to obtain the same value from the game with or without the experiment X. The value of the game is therefore unchanged; hence, the attainable minimax value is unchanged with the introduction of the experiment. Conversely, suppose there are two primary acts and that after the introduction of the experiment, nature has a strategy n that can ensure the same value of the game as before. Consider a minimal such strategy, in the sense that no strict subset of states played by nature with positive probability can be used by themselves to attain the same value. Nature could have played n in the game without the experiment, obtaining a value at least as high; therefore, this strategy is part of an equilibrium strategy pro le ( d; n) in the game without the experiment. However, if there exist !1; !2 2 and S 2 S with p1 = P(X 2 S j!1) 6= P(X 2 S j!2) = p2; then the decision maker can obtain a strictly higher payo if nature plays n after the introduction of the experiment. To see why, rst suppose that n is minmal in the game without the experiment. Given there are two primary acts f1 and f2, we can relabel !1 and !2 such that the payo from f1 is at least as high as that from f2 in state !1, and the payo from f2 is at least as high as that from f1 in state !2; further, one of these inequalities is strict. If this is not the case, then n is clearly not minimal (either !1 or !2 could be removed from the support of nature?s strategy). We can partition into 1 and 2, with !1 2 1, !2 2 2, and P(X 2 S j! 2 1) 6= P(X 2 S j! 2 2): Without loss of generality, suppose the rst probability is greater. Consider the strategy (X)d which plays f1 if X 2 S and f2 if X 2 Sc, and note that for small enough the strategy (X) d + (1 ) d will obtain a strictly higher payo than d in response to n. Alternatively, if n is no longer minimal when the experiment is removed from the game, we can nd a minimal equilibrium strategy 0n for nature to follow in the game without the experiment, which is supported on a strict subset of the states supporting n. By the argument in the preceding paragraph, we cannot have !1; !2 2 0 and S 2 S with p1 = P(X 2 S j!1) 6= P(X 2 S j!2) = p2; otherwise the experiment allows the decision maker to make a strict improvement. However, by the argument in the rst paragraph of this proof, this means that 0n is an equilibrium strategy of nature in the game with the experiment, guaranteeing the same payo . This contradicts the assumption that n was minimal in the game with the experiment. I note in passing that the restriction to situations with two primary acts in the second part of the proposition is needed to rule out the situation depicted in the following regret table, where many (but not all!) informative experiments would be insu cient to cause a minimax agent to deviate from f3. !1 !2 f1 0 10 f2 10 0 f3 1 1 The key insight of this result is that an experiment will be worthless to a minimax agent if it does not discriminate between the states that nature plays in equilibrium. Put this way, it sounds rather unremarkable, but it explains the unwillingness of minimax agents to pay for information in the examples given previously. If nature makes use of only a small subset of available states in equilibrium, then the minimax agent?s decision process is entirely dominated by this subset of states, and experiments giving information about 5 other states will have no value. In Savage?s example, nature has a dominant strategy in the negative income game, and consequently there can never be an experiment with value to a minimax negative income agent. In Parmigiani?s example, state RR is weakly dominated in the regret game, but not in the negative income game. Parmigiani (1992) proves a similar result under somewhat stronger conditions, and calls a decision rule ultrapessimistic about the experiment X if X is \relevant," yet has no value. An experiment is said to be relevant if there exist distinct states !1 and !2, and a set S 2 S, such that P(X 2 S j!1) 6= P(X 2 S j!2): As a corollary, it is shown that as long as there are at least three states, and nature has an equilibrium strategy that is supported on a strict subset of the state space, then there exist experiments for which the minimax rule is ultrapessimistic. The need for three distinct states stems from a feature of games in regret form: so long as the decision maker does not have a dominant strategy, nature must employ a mixed strategy in equilibrium. Thus, nature?s equilibrium strategy is supported on at least two states, and ultrapessimism cannot appear in the regret form of a decision problem with two states. This hints at a more signi cant distinction between minimax negative income and minimax regret. In the negative income game, nature may have strictly dominated strategies, and information in favor of these states will always be worthless. In a regret form game, no state ever represents a strictly dominated strategy for nature. This ensures that valuable experiments always exist, and further, a strong enough signal in favor of a given state will decrease the regret obtained in that state. Proposition 2. Consider a decision problem with nitely many states and nitely many primary acts. If the minimax regret is greater than zero, there exists an experiment with strictly positive value to a minimax regret agent. Note that the experiment need not reveal the true state with certainty. Further, Savage?s (1954) analysis of partition problems demonstrates that repeated observation will cause the minimax regret to approach zero. Proof. Let L denote the maximum regret from any act in any state, and let L denote the minimax regret of the decision problem in the absence of an experiment. Label the states of nature !1; !2; :::; !n, and let X be such that P(X = i j!i) = L L L ; P(X = i j!j ; j 6= i) = L (n 1)L : It is a simple exercise to verify that this experiment enables a strict improvement in minimax regret, and the signal i must increase the probability that the optimal action in state i is chosen. I contend that the spirit of Savage?s objection to minimax negative income lies in the existence of decision problems for which no such experiment can be devised, and in this sense, the critique clearly does not apply to the minimax regret decision rule. Indeed this claim is bolstered by the observation that a Bayesian decision maker is vulnerable to the charge of ultrapessimism as de ned above. Recall the payo table from Savage?s example. B1 B2 f1 1 1 f2 10 1 Given the same experiment X described in section 2, a Bayesian with a prior that assigns equal probability to B1 and B2 will place zero value on the experiment so long as g 211 . Clearly, experiments with 2 11 g < 1 2 are relevant according to the de nition given by Parmigiani (1992), so this would have us label the Bayesian as ultrapessimistic. Further, recall from our previous analysis that for this range of g, a minimax regret agent attaches a strictly positive value to the experiment. The existence of \relevant" but worthless experiments 6 for a minimax regret agent fails to distinguish minimax regret from other decision rules. The existence of decision problems where no experiment is valuable stands out as a property of minimax applied to negative income. 4 Discussion The analysis from the previous section suggests that the relevance of an experiment to a decision maker is best viewed in the context of the decision rule being applied: we could simply say that an observation is relevant if it has value to the decision maker, given the decision rule. The real question becomes what observations are relevant. For any pair of decision rules we have considered, examples can be constructed where an observation is relevant to one but not the other, or vice versa. Minimax applied to negative income has the peculiar feature of ignoring any conceivable experiment in some decision problems. Other examples of this are presented by Hodges and Lehmann (1950) and by Radner and Marschak (1954). Hodges and Lehmann examine the problem of estimating the parameter of a binomial distribution. Under one loss function they consider, the minimax negative income estimate is constant despite any observations made. Similarly, Radner and Marschak posit a hypothetical gamble on the outcome of ipping a weighted coin. They show that a decision maker applying a minimax negative income rule never observes more than one ip of the coin to obtain information, even if doing so is nearly costless. More recently, Manski (2004) has noted this problem in the context of treatment choice. In all of these examples, the minimax regret rule avoids the pitfall of failing to utilize relevant information; Savage (1954, p. 200) even addresses the example of Hodges and Lehmann (1950) directly. The Hurwicz criterion has been proposed as a way to address the pessimism of minimax applied to negative income. In its general form, the criterion suggests choosing an action f to maximize H(f) = (sup Bi I(f ;Bi); inf Bi I(f ;Bi)) where is any monotonically increasing function of both arguments, and fBig is a partition of the state space. Unfortunately, as shown by Radner and Marschak (1954), the Hurwicz criterion falls victim to essentially the same problem as the minimax negative income rule. A small set of states can entirely dominate the decision making process, and there are examples where no conceivable experiment is relevant. In the simplest case, suppose there are three possible events, B1, B2, and B3. Suppose that if B1 obtains, then no matter what action is chosen, the payo will be higher than in all other events, and suppose that if B3 obtains, then no matter what action is chosen, the payo will be lower than in all other events. The Hurwicz criterion then simpli es to choosing an action which maximizes H(f) = (I(f ;B1); I(f ;B3)); and the consequences of an action in event B2 are completely ignored, even if available information strongly suggests that B2 obtains. In the context of treatment choice, Stoye (2009) has renewed concerns about the pessimism of minimax applied to regret. Speci cally, he examines the way minimax regret makes use of the information provided by covariates, nding that in general, minimax regret is achieved by examining each value of the covariates separately. Pooling to any extent is never optimal by this criterion. Stoye?s Proposition 4 asserts that with a large enough covariate space|in particular, covariates taking uncountably many values|minimax regret will recommend a \no-data" rule. However, this result should be unsurprising. Given the state space under consideration, nature is free to independently choose a di erent outcome distribution for uncountably many values of the covariate. Under these circumstances, a Bayesian estimator can be shown to be inconsistent essentially everywhere1. A nite sample cannot provide meaningful information without some restriction 1Consistency in this context means that the estimator will converge in probability to the true parameter value. A pair comprised of a prior distribution on the parameter space, and a true value of the parameter, is consistent if that particular prior will converge in probability to that particular parameter value as the amount of available data increases. Freedman (1965) gives 7 of the state space. This example hardly constitutes a criticism of minimax regret; rather, it acknowledges fundamental limitations of statistical inference. Stoye?s broader critique relates to the implicit assumption made by the minimax regret procedure that the outcome distributions are independent across each value of the covariates. I must also object to this criticism as the appeal of his argument rests on our intuition that such distributions should be related. It strikes me as improper to critique a procedure designed to eliminate the use of subjective prior information on the basis of its disagreement with subjective prior information. Stoye (2009) has himself suggested the possibility of incorporating such information into the minimax regret procedure by suitably restricting the state space under consideration. I emphasize that I do not wish to dismiss these concerns regarding minimax applied to regret. Indeed, they represent important challenges that must be addressed in any application of the procedure. My point is that far from being fundamental aws, these issues represent a cautionary note on the proper implementation of minimax regret. Savage (1954, p. 203) acknowledged a distinct but related criticism of minimax decision rules, derived from another example given by Hodges and Lehmann (1950); the example again concerns estimating the parameter of a binomial distribution. If squared error is used as the loss function, then the minimax regret estimate su ers higher loss than the maximum likelihood estimate over most of the parameter space. As the number of trials grows without bound, the region in which minimax regret is superior becomes vanishingly small. In this example, minimax appears overly concerned with the possibility that the parameter lies in a small neighborhood of 12 , and it sacri ces performance elsewhere to reduce the risk of loss in this part of the parameter space. This phenomenon could also be described as pessimism. Stoye (2009) notes what I believe to be a related example in his concluding remarks, where a sample that is heavily skewed towards one treatment leads to a counter-intuitive recommendation. This may be an inevitable consequence of ensuring \uniform" performance across all possible states. In the case of the binomial distribution, it appears to be a result of the relatively high variance of a binomial random variable with parameter close to 12 . This feature of the binomial distribution renders it more di cult to estimate the parameter in this region, and obtaining uniform performance necessitates sacri cing performance in regions with lower variance. One might consider rescaling the loss function to obtain a more desirable estimate. Indeed, the choice of loss function for parameter estimation seems to have received little attention, yet it is fundamental to the behavior of the minimax regret procedure. With modern computational tools, it may be feasible to consider minimax regret estimates over a range of loss functions, providing another potential avenue for future research. References Freedman, D. (1965), \On the Asymptotic Behavior of Bayes Estimates in the Discrete Case II." The Annals of Mathematical Statistics, 36, 454{456. Hodges, J. and E. Lehmann (1950), \Some Problems in Minimax Point Estimation." Annals of Mathematical Statistics, 21, 182{197. Manski, C. (2004), \Statistical Treatment Rules for Heterogeneous Populations." Econometrica, 72, 1221{ 1246. Parasarathy, K. (1967), Probability Measures on Metric Spaces. Academic Press. Parmigiani, G. (1992), \Minimax, Information, and Ultrapessimism." Theory and Decision, 33, 241{252. an example with countably many parameters where the set of consistent prior-parameter pairs is of category 1. More generally, as a consequence of well-known results in measure theory (See Parasarathy, 1967, p. 29) and topology (See Schaefer, 1966, p. 23), all measures on complete, separable metric spaces are tight, and hence put probability one on a countable union of compact sets. Moreover, all locally compact Hausdor topological vector spaces are nite-dimensional. This implies that all measures on an in nite-dimensional Hausdor topological vector space assign probability one to a category 1 (\meagre") set. Therefore, with in nitely many parameters, a Bayesian estimator is inconsistent outside an insigni cant portion of the parameter space. 8 Radner, Roy and Jacob Marschak (1954), \Note on Some Proposed Decision Criteria." In Decision Processes (R. Thrall, C. Coombs, and R. Davis, eds.), 61{69, John Wiley and Sons. Savage, L. (1954), The Foundations of Statistics, second edition. Dover Publications Inc. 2nd Ed. published 1972. Schaefer, H. (1966), Topological Vector Spaces. MacMillan. Stoye, J. (2009), \Minimax Regret Treatment Choice with Finite Samples." Journal of Econometrics, 151, 70{81. Wald, A. (1950), Statistical Decision Functions. John Wiley and Sons. 9